fix(certbot): reissue when the configured domains change - #1137

Problem

Editing domains in certbot.toml does nothing once a certificate exists. The
operator adds a name, restarts certbot, and every run reports the certificate as
current:

INFO certbot::bot: checking if certificate needs to be renewed
INFO certbot::bot: certificate /e2e/wd/live/cert.pem is up to date

Two places have to agree on the name list, and neither consults the
configuration:

// create_cert_if_needed: existence is the whole test -- the names are never looked at
if live_cert_pem_path.as_ref().exists() && live_key_pem_path.as_ref().exists() {
    return Ok(false);
}

// renew_cert: the names come from the certificate being replaced
let domains = extract_subject_alt_names(cert_pem)?;
self.request_new_certificate(key_pem, &domains).await

So the name list is pinned at whatever the first issuance used, and renewal
copies it forward. The configuration edit is not deferred until expiry -- it
never takes effect at all, and nothing in the log says so.

This is the CLI's issuance path (certbot/src/bot.rs, used by
certbot/cli), which is how self-hosted deployments obtain the gateway
certificate -- docs/deployment.md and docs/dstack-gateway.md both drive it
through certbot.toml. The in-CVM gateway path is unaffected: DistributedCertBot
computes *.{domain} from its config on every issuance and never calls
renew_cert.

Fix

The live certificate is only the certificate that was asked for if it carries
the configured names, so create_cert_if_needed compares them and reissues on a
mismatch:

match extract_subject_alt_names(&live_cert_pem) {
    Ok(names) if names_match(&names, domains) => return Ok(false),
    Ok(names) => info!("reissuing: the live certificate covers {}, the configuration asks for {}", ...),
    Err(err) => warn!("reissuing: cannot read the live certificate's names: {err:#}"),
}

names_match compares sets, not sequences: the CA returns names in its own
order (the staging CA puts the wildcard first), DNS names are case-insensitive,
and a trailing root dot is the same name. Equality rather than containment, so
narrowing the list reissues too -- a certificate covering more than the
configuration asks for is not the configured certificate either.

An unreadable live certificate reissues rather than being left alone: it cannot
be checked against the configuration, so continuing to serve it means serving
something this process can no longer reason about. Issuance replaces it.

renew_cert still derives its names from the certificate it replaces, which is
correct once the certificate itself is kept in step: create_cert_if_needed
runs first on every cycle, so renewal only ever sees an aligned certificate.
The existing key is reused, as renewal already does.

A reissue that fails must not take the renewal down with it. The configuration
can name something the CA will never validate -- a typo, a zone the Cloudflare
token cannot write -- and that reissue then fails on every cycle. Failing the
run at that point would skip the renewal check below it, so one name the
operator got wrong would stop renewing the certificate that is actually being
served, until it expires 90 days later: a worse outcome than the edit being
ignored, which is what this PR started from. renew_inner reports the failure
and carries on to the renewal, and returns the failure only if the renewal
committed nothing of its own -- so a daemon keeps the deployment alive, a
renew --once still exits non-zero, and a renewal that did commit still runs
the renewed_hook. Nothing to protect on a first run, where there is no live
certificate: the error propagates as before.

The decision itself is reissue_reason(live_cert_pem, domains) -> Option<String>,
a pure function of the certificate and the configuration, so all three of its
outcomes are unit-testable. A live certificate that cannot even be read off disk
takes the same path as one that cannot be parsed, rather than failing the run.

Verification

Pebble + a mock Cloudflare API. Same workdir, domains edited between the two
runs, origin/next (55021edbf8) and this branch:

origin/next ends with one certificate in the backup directory, this branch
with two. The reissue announces itself:

INFO certbot::acme_client: reissuing: the live certificate covers sanchange.e2e.test, the configuration asks for sanchange.e2e.test, extra.e2e.test
INFO certbot::bot: created new certificate

Two distinct names rather than a name plus its wildcard, deliberately: a
base-plus-wildcard order fails on next for an unrelated reason, fixed
separately in #1136.

Unit tests cover names_match (reordering and case are not changes; an added or
dropped name is) and reissue_reason against real self-signed certificates (the
configured one is kept, a missing name reissues and names both lists, an
unparseable certificate reissues). cargo fmt, cargo clippy -p certbot -p certbot-cli --all-targets -D warnings and cargo test -p certbot are clean.

Two adjacent fixes

Found while documenting the CLI, both in separate commits so they can be dropped
independently.

certbot cfg walked the serialized document and the struct's doc comments in
lockstep by position, and a None option serializes to nothing -- so with
cf_api_url absent, every comment from there on described the key above the one
it belonged to:

# Optional Cloudflare-compatible API base URL
dns_txt_ttl = 60
# TTL for DNS TXT challenge records in seconds
auto_set_caa = true

The comment is now looked up by key name. init's help said it initializes the
configuration file; it reads the configuration file and creates the ACME
account.

The CLI also gets a README saying what it is for: it keeps the ACME account key
and the certificate key in a plain directory on the host that runs it, so
nothing about the issuance is attested, while a deployed gateway issues its own
certificates from the configuration it already holds and publishes the public
keys for ct_monitor to check.

The deployment guide was pointing at the CLI

docs/deployment.md step 4 told the operator to set GATEWAY_CERT/GATEWAY_KEY
in build-config.sh and run ./certbot renew -c certbot.toml on the host. A
gateway deployed by that guide runs as a CVM and issues its own certificates
from its admin config, keeping the ACME account key and every certificate in the
CVM's WaveKV store -- so the old text produced a certificate on the host that
nothing reads, from a file that is not part of the guide's flow.

The step now describes what step 3's bootstrap-cluster.sh already configured
(SetCertbotConfig, CreateDnsCredential, AddZtDomain), how to watch
issuance land in ListZtDomains, how SetCaa pins issuance to the gateway's
own ACME account, and what switching from staging to production takes. It moves
to the required part of the checklist: without a certificate for the domain the
gateway cannot serve an app over TLS.

docs/dstack-gateway.md had the same problem in the other deployment shape: it
walked a host-mode gateway through certbot.toml, ./certbot set-caa && ./certbot renew, and cert_chain/cert_key. A gateway on the host links the same crate
and issues from its own process just as one in a CVM does, so that guide now
configures the gateway -- the proxy fields, the admin API the ACME settings are
stored through, and a data_dir that survives restarts -- and then walks the
same three admin RPCs. cert_chain/cert_key are documented for what they are:
a certificate something else produced, loaded at startup.

Two things noticed in that file while editing it: core.admin.auth_token is the
current key name (the guide used the accepted older admin_token), and the
alerting table listed dstack_gateway_kv_persist_failures_total twice.